Optimization and Nonlinear Equations

Gordon K. Smyth May 1997 Optimization means to nd that value of x which maximizes or minimizes a given function f(x). The idea of optimization goes to the heart of statistical methodology, as it is involved in solving statistical problems based on least squares, maximum likelihood, posterior mode and so on. A closely related problem is that of solving a nonlinear equation, g(x) = 0 for x where g is a possibly multivariate function. Many algorithms for minimizing f(x) are in fact derived from algorithms for solving g = @f=@x = 0, where @f=@x is the vector of derivatives of f with respect to the components of x. Except in linear cases, optimization and equation solving invariably proceed by iteration. Starting from an approximate trial solution, a useful algorithm will gradually re ne the working estimate until a pre-determined level of precision has been reached. If the functions are smooth, a good algorithm can be expected to converge to a solution when given a su ciently good starting value. A good starting value is one of the keys to success. In general, nding a starting value requires heuristics and an analysis of the problem. One strategy for tting complex statistical models, by maximum likelihood or otherwise, is to progress from the simple to the complex in stages. Fit a series of models of increasing complexity, using the simpler model as a starting value for the more complicated model in each case. Maximum likelihood iterations can often be initialized by using a less e cient moment estimator. In some special cases, such as generalized linear models, it is possible to use the data itself as a starting value for the tted values. An extremum (maximumor minimum) of f can be either global (truly the extreme value of f over its range) or local (the extreme value of f in a neighborhood containing the value). (See Figure 1.) Generally it is the global extremum that we want. (A maximum likelihood estimator, for example, is by de nition the global maximum of the likelihood.) Unfortunately, distinguishing local extrema from the global extremum is not an easy task. One heuristic is to start the iteration from several widely varying starting points, and to take the most extreme (if they are not equal). If necessary a large number of starting values can be randomly generated. Another heuristic is to perturb a local extremum slightly to check that the algorithm returns to it. A relaD x1 x2 x f(x)